Skip to content

perf: bypass Arrow for asyncpg record COPY - #666

Merged
cofin merged 2 commits into
mainfrom
feat/asyncpg-direct-record-copy
Aug 1, 2026
Merged

perf: bypass Arrow for asyncpg record COPY#666
cofin merged 2 commits into
mainfrom
feat/asyncpg-direct-record-copy

Conversation

@cofin

@cofin cofin commented Aug 1, 2026

Copy link
Copy Markdown
Member

Summary

  • bypass the generic records-to-Arrow-to-records round trip in AsyncpgDriver.load_from_records()
  • validate and normalize mapping or positional records directly into ordered tuples for one binary COPY call
  • share safe quoted and schema-qualified COPY target handling between record and Arrow ingestion
  • retain the existing Arrow coercion and row conversion path for actual Arrow inputs
  • document the AsyncPG-specific direct-record path and add a conditional COPY-versus-executemany() crossover benchmark

Root cause

The generic load_from_records() implementation materialized every in-memory batch as a PyArrow table. AsyncPG's native Arrow ingest implementation then converted that table back into Python row tuples before calling copy_records_to_table(). The redundant conversion imposed fixed overhead that dominated small and medium record batches.

Behavior

Direct AsyncPG record loads now preserve explicit column order or the first mapping's key order, validate the full batch before truncating or copying, and retain native JSON/JSONB mappings, arrays, UUIDs, and nulls. Empty batches, mixed row shapes, inconsistent mapping keys, missing positional columns, and row-width mismatches fail before database I/O.

Record batches execute exactly one copy_records_to_table() call. Returned jobs report the original destination, format="records", exact rows_processed, and bytes_processed=0. The inherited CockroachDB driver uses the same implementation without changing its advertised capabilities.

Benchmark

The conditional real-PostgreSQL benchmark uses one session and table, three warmups, truncation outside timed sections, and 15 timed samples per variant. Median results from the local PostgreSQL service were:

Rows Direct COPY executemany()
1 924.5 rows/s 1,677.6 rows/s
2 1,679.8 rows/s 3,613.9 rows/s
10 7,292.2 rows/s 17,049.7 rows/s
100 77,388.9 rows/s 88,607.6 rows/s
1,000 251,331.9 rows/s 157,050.1 rows/s
5,000 465,732.6 rows/s 186,891.7 rows/s

Closes #664

@codecov-commenter

codecov-commenter commented Aug 1, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.59155% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 77.27%. Comparing base (ce2b042) to head (76f950b).

Files with missing lines Patch % Lines
sqlspec/adapters/asyncpg/driver.py 98.59% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #666      +/-   ##
==========================================
+ Coverage   77.22%   77.27%   +0.05%     
==========================================
  Files         475      475              
  Lines       67824    67892      +68     
  Branches     9322     9335      +13     
==========================================
+ Hits        52374    52463      +89     
+ Misses      12042    12038       -4     
+ Partials     3408     3391      -17     
Flag Coverage Δ
py3.10 75.55% <98.59%> (+0.03%) ⬆️
py3.11 75.57% <98.59%> (+0.03%) ⬆️
py3.12 75.57% <98.59%> (+0.03%) ⬆️
py3.13 75.56% <98.59%> (+0.03%) ⬆️
py3.14 76.45% <98.59%> (+0.02%) ⬆️
unit 64.78% <98.59%> (+0.05%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
sqlspec/adapters/asyncpg/driver.py 90.34% <98.59%> (+8.12%) ⬆️

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@cofin
cofin merged commit 16aa5fb into main Aug 1, 2026
23 checks passed
@cofin
cofin deleted the feat/asyncpg-direct-record-copy branch August 1, 2026 22:27
@cofin cofin changed the title perf(#664): bypass Arrow for asyncpg record COPY perf: bypass Arrow for asyncpg record COPY Aug 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Optimize asyncpg load_from_records by bypassing Arrow rematerialization

2 participants